Three Algorithms for Cholesky Factorization on Distributed Memory Using Packed Storage

We present three algorithms for Cholesky factorization using minimum block storage for a distributed memory (DM) environment. One of the distributed square blocked packed (SBP) format algorithms performs similar to ScaLAPACK PDPOTRF, and with iteration overlapping outperforms it by as much as 67%. By storing the blocks in a standard contiguous way, we get better performing BLAS operations. Our DM algorithms are almost insensitive to memory hierarchy effects and thus gives smooth and predictable performance. We investigate the intricacies of using RFP format in a DM ScaLAPACK environment and point out some advantages and drawbacks. 1 Near Minimal Storage in a Serial Environment Rectangular full packed (RFP) format is a standard full storage two-dimensional array for triangular or symmetric matrices requiring minimum storage [3]. For the lower triangular case, blocks A11, A21, A22 are stored as submatrices in a rectangular full storage array. This allows for using level 3 BLAS as well as making it easy to write LAPACK-style code for this format [3]. SBP format is a generalization of standard full storage. The matrix is partitioned into square blocks of order NB, and in the case of storing symmetric or triangular matrices, only the triangular blocks are stored. Each square block is contiguous in memory; the blocks are either stored row or column-wise. Each square diagonal block wastes NB(NB-1)/2 elements, a total of N(NB-1)/2 elements summed over all N/NB diagonal blocks. Each square block will map into L1 cache in an optimal way resulting in efficient BLAS operations. 2 Minimum Block Storage in a Distributed Environment The current industry standard for distributed memory computing views the processors as a PxQ mesh and uses a 2D Block Cyclic Layout (BCL) of full format arrays. This has proven to be a good choice for achieving effective load balancing. However, this wastes about half the storage for triangular and symmetric matrices. There is currently no industry standard for packed storage. The SBP storage of Section 1 is a possibility.